{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to create a numeric calculation web service that uses autoscaling GPUs\n",
    "\n",
    "This notebook demonstrates how to create a web service that carries out numerical computations on a GPU.\n",
    "It uses the following capabilities:\n",
    "* Keras/TensorFlow models automatically run on a GPU if found\n",
    "* TensorFlow provides an easy wrapper for most numpy functions\n",
    "* It's possible to specify a custom serving function for Keras models\n",
    "* Cloud AI Platform provides a easy way to deploy models as a web service\n",
    "\n",
    "It accompanies this blog post:\n",
    "https://medium.com/@lakshmanok/how-to-create-a-numeric-calculation-web-service-that-uses-autoscaling-gpus-c43b865d867d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Original function\n",
    "As an example, let's use a function that calculates the time of sunrise/sunset given the latitude, longitude of a point\n",
    "\n",
    "Note how I'm aliasing sin, cos, etc. to the corresponding TensorFlow functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op Mul in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op RealDiv in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Sin in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Cos in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Sub in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Asin in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Tan in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Neg in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Acos in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op AddV2 in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Floor in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "{'dayNo': 15.0, 'sunrise_hr': <tf.Tensor: shape=(), dtype=float32, numpy=7.0>, 'sunrise_min': <tf.Tensor: shape=(), dtype=float32, numpy=40.322144>, 'sunset_hr': <tf.Tensor: shape=(), dtype=float32, numpy=17.0>, 'sunset_min': <tf.Tensor: shape=(), dtype=float32, numpy=9.641876>}\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import math\n",
    "import tensorflow as tf\n",
    "\n",
    "tf.debugging.set_log_device_placement(True)\n",
    "\n",
    "def calc_sunrise_sunset(lat: float, lng: float, dayNo: int, utcOffset: int):\n",
    "    \"\"\"\n",
    "    Specify the location of the point, the day you are interested in (0 is Jan. 1)\n",
    "    https://math.stackexchange.com/questions/2186683/how-to-calculate-sunrise-and-sunset-times/2199903#2199903\n",
    "    \"\"\"\n",
    "    # aliases\n",
    "    pi = tf.constant(math.pi)\n",
    "    sin, cos, asin, acos, tan, floor = tf.math.sin, tf.math.cos, tf.math.asin, tf.math.acos, tf.math.tan, tf.math.floor\n",
    "    \n",
    "    # actual calc, without any atmospheric correction\n",
    "    longCorr = 4*(lng - 15*utcOffset);\n",
    "    B = 2*pi*(dayNo - 81)/365;\n",
    "    EoTCorr = 9.87*sin(2*B) - 7.53*cos(B) - 1.5*sin(B);\n",
    "    solarCorr = longCorr - EoTCorr;\n",
    "    delta = asin(sin(23.45 * pi/180)*sin(2*pi*(dayNo - 81)/365));\n",
    "    sunrise = 12 - (180/pi)*acos(-tan(lat * pi/180)*tan(delta))/15 - solarCorr/60;\n",
    "    sunset  = 12 + (180/pi)*acos(-tan(lat * pi/180)*tan(delta))/15 - solarCorr/60;\n",
    "    \n",
    "    sunrise_hr = floor(sunrise)\n",
    "    sunrise_min = 60 * (sunrise - sunrise_hr)\n",
    "    sunset_hr = floor(sunset)\n",
    "    sunset_min = 60 * (sunset - sunset_hr)\n",
    "    \n",
    "    return {\n",
    "        'dayNo': dayNo,\n",
    "        'sunrise_hr': sunrise_hr,\n",
    "        'sunrise_min': sunrise_min,\n",
    "        'sunset_hr': sunset_hr,\n",
    "        'sunset_min': sunset_min\n",
    "    }\n",
    "    \n",
    "print(calc_sunrise_sunset(39.833, -98.583, 15, -6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keras Model\n",
    "\n",
    "Create a no-op Keras Model and export it with the above serving function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Add in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op VarIsInitializedOp in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op LogicalNot in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Assert in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Reshape in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1786: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "Executing op StringJoin in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ShardedFilename in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op SaveV2 in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Pack in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op MergeV2Checkpoints in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op Identity in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "INFO:tensorflow:Assets written to: export/sunrise_20200927_004422/assets\n"
     ]
    }
   ],
   "source": [
    "import os, datetime, shutil\n",
    "\n",
    "@tf.function(input_signature=[\n",
    "    tf.TensorSpec([None], dtype=tf.float32),\n",
    "    tf.TensorSpec([None], dtype=tf.float32),\n",
    "    tf.TensorSpec([None], dtype=tf.float32),\n",
    "    tf.TensorSpec([None], dtype=tf.float32),\n",
    "])\n",
    "def calc_on_gpu(lat, lng, dayno, utc_offset):\n",
    "    return calc_sunrise_sunset(lat, lng, dayno, utc_offset)\n",
    "\n",
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Dense(1, input_shape=[1])\n",
    "  ])\n",
    "\n",
    "shutil.rmtree('export', ignore_errors=True)\n",
    "export_path = os.path.join('export', 'sunrise_{}'.format(datetime.datetime.now().strftime(\"%Y%m%d_%H%M%S\")))\n",
    "model.save(export_path, signatures={'serving_default': calc_on_gpu})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The given SavedModel SignatureDef contains the following input(s):\n",
      "  inputs['dayno'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1)\n",
      "      name: serving_default_dayno:0\n",
      "  inputs['lat'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1)\n",
      "      name: serving_default_lat:0\n",
      "  inputs['lng'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1)\n",
      "      name: serving_default_lng:0\n",
      "  inputs['utc_offset'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1)\n",
      "      name: serving_default_utc_offset:0\n",
      "The given SavedModel SignatureDef contains the following output(s):\n",
      "  outputs['dayNo'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1)\n",
      "      name: PartitionedCall:0\n",
      "  outputs['sunrise_hr'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1)\n",
      "      name: PartitionedCall:1\n",
      "  outputs['sunrise_min'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1)\n",
      "      name: PartitionedCall:2\n",
      "  outputs['sunset_hr'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1)\n",
      "      name: PartitionedCall:3\n",
      "  outputs['sunset_min'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1)\n",
      "      name: PartitionedCall:4\n",
      "Method name is: tensorflow/serving/predict\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show --dir {export_path} --tag_set serve --signature_def serving_default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op RestoreV2 in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op RestoreV2 in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op __inference_signature_wrapper_554 in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "{'sunrise_min': <tf.Tensor: shape=(), dtype=float32, numpy=40.322113>, 'sunset_hr': <tf.Tensor: shape=(), dtype=float32, numpy=17.0>, 'sunset_min': <tf.Tensor: shape=(), dtype=float32, numpy=9.641991>, 'sunrise_hr': <tf.Tensor: shape=(), dtype=float32, numpy=7.0>, 'dayNo': <tf.Tensor: shape=(), dtype=float32, numpy=15.0>}\n"
     ]
    }
   ],
   "source": [
    "restored = tf.keras.models.load_model(export_path)\n",
    "infer = restored.signatures['serving_default']\n",
    "# note input name\n",
    "outputs = infer(lat=tf.constant(39.833), lng=tf.constant(-98.583), dayno=tf.constant(15.0), utc_offset=tf.constant(-6.0))\n",
    "print(outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy to Cloud AI Platform Predictions\n",
    "\n",
    "We can deploy the model to AI Platform Predictions which will take care of scaling.\n",
    "The key line here is:\n",
    "```\n",
    "       --machine-type n1-standard-2 --accelerator count=1,type=nvidia-tesla-k80\n",
    "```\n",
    "This will deploy to a machine with 2 CPUs and 1 GPU.\n",
    "\n",
    "For more details, see: https://cloud.google.com/ai-platform/prediction/docs/machine-types-online-prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "export/sunrise_20200927_004422\n"
     ]
    }
   ],
   "source": [
    "!find export/ | head -2 | tail -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sunrise\n",
      "Model sunrise already exists\n",
      "\n",
      "Creating version v1 from export/sunrise_20200927_004422\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using endpoint [https://ml.googleapis.com/]\n",
      "Using endpoint [https://ml.googleapis.com/]\n",
      "Listed 0 items.\n",
      "Using endpoint [https://ml.googleapis.com/]\n",
      "Creating version (this might take a few minutes)......\n",
      "....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "MODEL_LOCATION=$(find export | head -2 | tail -1)\n",
    "MODEL_NAME=sunrise\n",
    "MODEL_VERSION=v1\n",
    "\n",
    "TFVERSION=2.1\n",
    "REGION=us-central1\n",
    "BUCKET=ai-analytics-solutions-kfpdemo\n",
    "\n",
    "# create the model if it doesn't already exist\n",
    "modelname=$(gcloud ai-platform models list | grep -w \"$MODEL_NAME\")\n",
    "echo $modelname\n",
    "if [ -z \"$modelname\" ]; then\n",
    "   echo \"Creating model $MODEL_NAME\"\n",
    "   gcloud ai-platform models create ${MODEL_NAME} --regions $REGION\n",
    "else\n",
    "   echo \"Model $MODEL_NAME already exists\"\n",
    "fi\n",
    "\n",
    "# delete the model version if it already exists\n",
    "modelver=$(gcloud ai-platform versions list --model \"$MODEL_NAME\" | grep -w \"$MODEL_VERSION\")\n",
    "echo $modelver\n",
    "if [ \"$modelver\" ]; then\n",
    "   echo \"Deleting version $MODEL_VERSION\"\n",
    "   yes | gcloud ai-platform versions delete ${MODEL_VERSION} --model ${MODEL_NAME}\n",
    "   sleep 10\n",
    "fi\n",
    "\n",
    "\n",
    "echo \"Creating version $MODEL_VERSION from $MODEL_LOCATION\"\n",
    "gcloud ai-platform versions create ${MODEL_VERSION} \\\n",
    "       --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --staging-bucket gs://${BUCKET} \\\n",
    "       --runtime-version $TFVERSION \\\n",
    "       --machine-type n1-standard-2 --accelerator count=1,type=nvidia-tesla-k80"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing input.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile input.json\n",
    "{\"lat\": 39.833, \"lng\": -98.583, \"dayno\": 15, \"utc_offset\": -6}\n",
    "{\"lat\": 39.833, \"lng\": -98.583, \"dayno\": 45, \"utc_offset\": -6}\n",
    "{\"lat\": 39.833, \"lng\": -98.583, \"dayno\": 72, \"utc_offset\": -6}\n",
    "{\"lat\": 39.833, \"lng\": -98.583, \"dayno\": 102, \"utc_offset\": -6}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using endpoint [https://ml.googleapis.com/]\n",
      "DAY_NO  SUNRISE_HR  SUNRISE_MIN  SUNSET_HR  SUNSET_MIN\n",
      "15.0    7.0         40.3221      17.0       9.64199\n",
      "45.0    7.0         5.45803      17.0       34.0227\n",
      "72.0    6.0         35.8807      18.0       12.3474\n",
      "102.0   6.0         6.04737      19.0       0.529518\n"
     ]
    }
   ],
   "source": [
    "!gcloud ai-platform predict --model sunrise --json-instances input.json --version v1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "response = {'predictions': [{'dayNo': 15.0, 'sunrise_min': 40.322113037109375, 'sunset_min': 9.641990661621094, 'sunrise_hr': 7.0, 'sunset_hr': 17.0}, {'dayNo': 45.0, 'sunrise_min': 5.458030700683594, 'sunset_min': 34.02271270751953, 'sunrise_hr': 7.0, 'sunset_hr': 17.0}, {'dayNo': 72.0, 'sunrise_min': 35.88074493408203, 'sunset_min': 12.347373962402344, 'sunrise_hr': 6.0, 'sunset_hr': 18.0}, {'dayNo': 102.0, 'sunrise_min': 6.047401428222656, 'sunset_min': 0.5295181274414062, 'sunrise_hr': 6.0, 'sunset_hr': 19.0}]}\n"
     ]
    }
   ],
   "source": [
    "from googleapiclient import discovery\n",
    "from oauth2client.client import GoogleCredentials\n",
    "import json\n",
    "\n",
    "credentials = GoogleCredentials.get_application_default()\n",
    "api = discovery.build(\"ml\", \"v1\", credentials = credentials,\n",
    "            discoveryServiceUrl = \"https://storage.googleapis.com/cloud-ml/discovery/ml_v1_discovery.json\")\n",
    "\n",
    "request_data = {\"instances\":\n",
    "  [\n",
    "    {\"lat\": 39.833, \"lng\": -98.583, \"dayno\": 15, \"utc_offset\": -6},\n",
    "    {\"lat\": 39.833, \"lng\": -98.583, \"dayno\": 45, \"utc_offset\": -6},\n",
    "    {\"lat\": 39.833, \"lng\": -98.583, \"dayno\": 72, \"utc_offset\": -6},\n",
    "    {\"lat\": 39.833, \"lng\": -98.583, \"dayno\": 102, \"utc_offset\": -6}\n",
    "  ]\n",
    "}\n",
    "\n",
    "parent = \"projects/{}/models/sunrise\".format(\"ai-analytics-solutions\", \"v1\") # use default version\n",
    "\n",
    "response = api.projects().predict(body = request_data, name = parent).execute()\n",
    "print(\"response = {0}\".format(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "40.322113037109375\n"
     ]
    }
   ],
   "source": [
    "print(response['predictions'][0]['sunrise_min'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete the web service\n",
    "\n",
    "The REST endpoint scales down to one node when it doesn't encounter any traffic. \n",
    "To avoid paying for that node, let's delete the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "v1 gs://ai-analytics-solutions-kfpdemo/207d95f93d36fe5ba780b9885c903f6a2d1ce78ab57bbd54e3cd32b59db022dc/ READY\n",
      "Deleting version v1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using endpoint [https://ml.googleapis.com/]\n",
      "Using endpoint [https://ml.googleapis.com/]\n",
      "This will delete version [v1]...\n",
      "\n",
      "Do you want to continue (Y/n)?  \n",
      "Deleting version [v1]......\n",
      "................................................................................done.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "MODEL_NAME=sunrise\n",
    "MODEL_VERSION=v1\n",
    "\n",
    "# delete the model version if it already exists\n",
    "modelver=$(gcloud ai-platform versions list --model \"$MODEL_NAME\" | grep -w \"$MODEL_VERSION\")\n",
    "echo $modelver\n",
    "if [ \"$modelver\" ]; then\n",
    "   echo \"Deleting version $MODEL_VERSION\"\n",
    "   yes | gcloud ai-platform versions delete ${MODEL_VERSION} --model ${MODEL_NAME}\n",
    "   sleep 10\n",
    "fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2020 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-gpu.2-1.m54",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-1:m54"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
